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Q ■ Abstract 

' We consider both channel coding and source coding, with perfect past feedback/feedforward, 

in the presence of side information. It is first observed that feedback does not increase 
the capacity of the Gel'fand-Pinsker channel, nor does feedforward improve the achiev- 
able rate-distortion performance in the Wyner-Ziv problem. We then focus on the 
Gaussian case showing that, as in the absence of side information, feedback/feedforward 
allows to efficiently attain the respective performance limits. In particular, we derive 
^ ' schemes via variations on that of Schalkwijk and Kailath. These variants, which are 

as simple as their origin and require no binning, are shown to achieve, respectively, the 
capacity of Costa's channel, and the Wyner-Ziv rate distortion function. Finally, we 
consider the finite-alphabet setting and derive schemes for both the channel and the 
' source coding problems that attain the fundamental limits, using variations on schemes 

! of Ahlswede and Ooi and Wornell, and of Martinian and Wornell, respectively. 
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1 Introduction 



That feedback does not increase the capacity of a memoryless channel, yet can dramatically 
simplify the schemes for achieving it, is a well known fact (cf. [6] and the literature survey 
therein). More recently, an analogous phenomenon was shown to hold for the dual problem 
of lossy source coding with perfect past feedback, aka 'feedforward', at the decoder [12, 7, 5], 
a problem arising in contexts as diverse as prediction theory, remote sensing, and control. 

In this work, wc revisit these problems to accommodate the presence of side informa- 
tion. As is the case for problems without feedback/feedforward, the only scenarios with 
fundamental limits, and achieving schemes, that are not directly implied from those known 
for the absence of side information are, respectively, the presence of side information only 
at the encoder, and only at the decoder, for the channel coding and the source coding 
problems. 

Our first observation in this context is that the fact that feedback/feedforward does 
not improve the fundamental performance limits carries over to these cases where side 
information is present. To see this, consider first channel coding for the Gel'fand-Pinsker 
channel [4] with feedback, by which we mean the following: The channel state information 

is available to the sender, and the memoryless channel has transition probability a:; , s) 
that depends on the input X and the state S. Si are assumed i.i.d.~ p{s). For a message 
index W G {1, 2, . . . , 2"^}, the i-th channel input is of the form Xi{W,S'',Y'-'^), i.e., 
allowed to depend on the past channel output symbols. Decoding is, as usual, based on the 
channel output Y"'. 

Observation 1 Feedback does not increase the capacity of the Gel'fand-Pinsker channel. 

Proof: One need merely observe that the original converse proof of Gcl'fand and Pinskcr 
[4] is general enough so as to include feedback. In other words, Ui — {Xi,Si) — Yi, where 
Ui = {W, S^^i,Y^~^), continues to be a Markov chain even in the presence of feedback. 
To see this, note that P{yi\w,s"-,y^~^) = p{yi\xi,Si) so {W, ,Y^~^) — {Xi,Si) — Yi is a 
Markov chain and therefore, since Ui is a deterministic function of (W, S", so is 

Ui-{Xi,Si)-Yi. Q.E.D. 

Though our interest in this work, and the schemes we develop, are for the case of non- 
causal state information, we mention in passing that a similar conclusion applies also for the 
Shannon channel with causal SI, where the i-th channel input is of the form X^iW, S^, 
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The independence between Ui and Si in the causal case is readily verified to persevere in 
the presence of feedback, implying: Feedback does not increase the capacity of the Shannon 
channel (with causal SI). 

Moving to the source coding analogue, consider the problem of Wyner-Ziv source cod- 
ing [14] with feedforward: The source and side information arc generated as independent 
drawings of the pair {Xi,Yi). Encoding, as in the original problem, is done by mapping 
the sequence X" into T G {1, 2, . . . , 2"^}. The i-th. reconstruction this time is of the form 
Xi{T,Y"',X'^~^), i.e., allowed to depend also on the past, non-quantized, past source sym- 
bols. This setting is the extension of the source coding with feedforward problem [12, 7, 5] 
to the case of side information at the decoder. 

Observation 2 Feedforward does not improve the rate distortion tradeoff in the Wyner- 
Ziv problem. 

Proof: Here too, the original converse proof carries over essentially unchanged. Specifi- 
cally, in the notation of [3, Section 14.9], we only need to add to Wj, resulting in 
Wi = (T, Y^~^,Y^-^,X^~^). The converse proof of [3, Theorem 14.9.1] carries over verbatim 
(erasing line (14.298) therein), since Wi — Xi — Yi continues to form a Markov chain under 
this modified Wi. Q.E.D. 

Given observations 1 and 2, it is natural to ask whether, similarly as in the absence of side 
information, feedback/feedforward can lead to simple schemes for attaining the fundamental 
limits. For the Gaussian case, we answer this question in the affirmative in the next section. 
More specifically, we present efficient schemes that exploit feedback/feedforward to achieve 
the capacity of Costa's channel [2], and the Wyner-Ziv function for a source which is a 
Gaussian-noise-corrupted version of the side information. Our schemes, which are variations 
on those of Schalkwijk and Kailath [9, 8], are as efficient as their origin and, in particular, do 
not require binning. In Section 3, we consider the finite alphabet setting and derive a scheme 
for the Gel'fand-Pinsker channel with feedback, building on the ideas of [1, 6]. We also 
derive a scheme for the dual problem of Wyner-Ziv coding with feedforward, by extending 
the approach of [5]. Our schemes for the finite-alphabet setting rely on Slepian-Wolf coding 
[11], and thus we make no claim at this point regarding the efficiency with which they 
can be implemented (in comparison to the efficiency of practical schemes for the Gel'fand- 
Pinsker channel and the Wyner-Ziv problem in the absence of feedback/feedforward). They 
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are, however, conceptually simple and suggest another view on the information-theoretic 
formulas of the Gel'fand-Pinsker capacity and the Wyner-Ziv rate-distortion function. 
They also shed light on yet another aspect of the duality between source coding and channel 
coding with side information. 

2 Variations on the Schalkwijk— Kailath Schemes 

A. Writing on Dirty Paper with On— line Proofreading: Costa's Channel 
with Feedback 

Consider the channel Yi = Xi + Si + Zi, where {Si} is an interference signal (with ESi = 
and cr| = ESf < oo) known to the encoder, and {Zi} is zero-mean, i.i.d. Gaussian noise 
with variance erf. Let the transmission power be limited to P. We now describe a modified 
version of the scheme of [8] for coding with feedback, which achieves the capacity C = 
^ log(l + P/(jf ). Moreover, for every R < C, the error probability is identical to that of 
the original scheme, as if Si were identically zero, namely, it decays double-exponentially 
rapidly with C — R. 



Initialization: Define a = y^l + -P/cr|, and g = P/u\. Given a message m = 0, 1, . . . , M — 
1, M = 2"-^, let e = {m + l/2)/M. Given 5" = {Si, . . . ,5„) define V2 = Si/a, and for 
i = 2, 3, . . . , n, compute recursively: 

1 \ Si 



\ ) or '-g 

Finally, let 6' = 9 + tpn+i- 

Recursion: For i = 1, set Xi^i = 0.5 and transmit a{Xi^i — 6'). At the receiver, compute 

X2^i = Xi^2 = ^1,1 ~ ^ and send X2^i back to the transmitter. For i = 2,3, . . . ,n, transmit 
a'^~^g{Xi^i — 6' + ^i). At the receiver, compute Xi^2 = — a'^^g ' then update 

and (for i < n) send X^^^i^^i back to the transmitter. 

Finally, decode m by quantizing to its message interval. 

Analysis: First, note that 

^ ^ a{Xi,i-e') + Si + Zi 
X2,l = 

a a 

= 0'-^2-—. (1) 

a 



We now argue that for all i >2, Xi^i = 9' — ipi — ^j, where {V'j} are defined as above, and 
{4>i} are defined hy ^2 = Z\/a and by the recursion 



4>: 



+ 1 



1 



2,3, ... ,n. 



We prove this by induction: For i = 2, this has been shown already in eq. (1). Assuming 
now that the hypothesis is true for a given i > 2, then 



—^^%,\ + ( 1 9 ) ^i.l 



a''^g{X,^x -9' + ^Pi) + Si + Zi 



Xi,i 
1 



a' 









Si 






1 ^ 


\ Zi 1 







Si + -^j 



-9^^ - 

confirming the induction hypothesis for i + 1. Thus, for i = n + 1, we get 

^{n+l),l = 0' — i'n+l — = 9 — (pn+l- 



(2) 



(3) 



But ^n+i is exactly the estimation error variable in [8], whose variance has been shown to be 
(rl/a^". Thus, the decision made by this scheme is identical to that of Schalkwijk's scheme 
(with S"" = 0) for every realization of the noise sequence. Obviously, the error performance 
is then the same too. 

As for the transmission power, we will distinguish again between i = 1 and i > 2. 
For z = 1, the transmission power is approximately a^(l/12 + Var{?/;„+i}), where 1/12 
approximates the variance of 9 as one corresponding to the uniform distribution in [0, 1] , 
and \&x{il)n+i} is bounded independently of n since V'n+i is a linear combination of {Si} 
with coefficients that decay exponentially with i. As for ? > 2, the transmission power is 



a'^^-')g^E{Xi,,-9' + ^if = a^^^''^ g' Ecf: 



(4) 
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where the second equahty has been proved in [8] (and can also easily be seen by induction, 
using the recursive definition of {0j})- Thus, except for i = 1, the transmission power is P 
at all times, which means that for large n the total average power tends to P. 
At the point, a few comments are in order: 

1. We have seen that in the presence of feedback, it is possible to achieve capacity with 
a simple scheme, without binning. 

2. While in the absence of feedback [2], the idea is not to 'fight' the interference by trying 
to pre-cancel it but rather to harness it to our own benefit, here the pre-cancelling 
approach seems to be fruitful. This is manifested both at the transmitter, where the 
contribution of {Si} to the estimation error to be transmitted is cancelled in order 
to save power, and in the definition of 9', which shifts 9 by an amount (V'n+i) which 
pre-cancels the contribution of {Si} to the error of the final estimator. 

3. As mentioned earlier, operatively, this scheme gives exactly the same estimation and 
decoding as in [8] for every realization of the noise process, and as if were non- 
existent (5" = 0). 

4. Similarly to the non-feedback case, the probability law of {Sn} is immaterial. The 
only requirement is that (7| < oo to assure that the expected power used at time i = 1 
is finite. 

5. Note that the non-causal dependence of the transmission on 5" is only via one number, 
B. A Scheme for Wyner— Ziv Coding with Feedforward 

Consider first rate distortion coding with feedforward in the absence of side information 
[12]. Let {Xi}\^^ be i.i.d. M{0,a^) and, for a given positive real f3, let 

I I 

k=2 

Let Y be the quantized version of Y using a uniform scalar quantizer on the interval 
[—A/2, A/2] with M levels (truncating values outside the interval). Encoder describes 
Y to decoder by giving index I{Y) of the quantization cell. Decoder reconstructs as follows: 
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Xi = -pY, ±2 = ^JW^{Xi - Xi), Xi = - (^2 _ for i = 3, . . . , Z. It 

was shown in [7] that, for Z > 1, 

-^E{Xi-X,) ^ + . (6) 

1=1 ' 

To see how this scheme attains the rate distortion function, fix the rate R (so M = 2^') 
and a small e > throughout. Take /? = 2^~'^^ and A = 2'*^. We note the following: 

1. E'k=2{P^ - 1)/?"^^*^+^^ < oo so the variance of Y is bounded (does not exceed a fixed 
value) regardless of I. 

2. Pr{y ^ [—A/2, A/2]} is diminishing with I (in fact, double-exponentially rapidly 
since Y is Gaussian with bounded variance and is A exponentially growing with /). 

3. In [—A/2, A/2] we are performing uniform quantization with resolution A/M = 

4. The two previous items imply that E{Y - Y)^ < c{A/M)^ = c2~2(^~^)' for an l- 
independent constant c (in fact, a high-resolution quantization argument will give the 
more refined E{Y - Y)^ ~ ^2-^^^-^'^'^) . 

5. Substituting into (6), we get, as I grows large, that the first term on the right side di- 
minishes, while the second one converges to ^ = a'^2~'^^^~'^^\ which is the distortion- 
rate function (up to the small e factor). 

Performance analysis for our scheme below will rely also on: 

Claim 1 The scheme described is robust in the sense that if the decoder receives any index 

I such that log \I{Y) — I\ = o{l), then the distortion converges, as for the original scheme, 
to a22-2(«-2£)_ 

Proof: The distance between the centers of two adjacent quantization cells is 2-(^-=)', so, 
letting Y denote the value off" that the decoder assumes based on /, |y— y| < 2~^^~^~^°^^^^^ . 
The error in reconstruction due to this discrepancy can increase from one component to 
the next by a factor of /3 = 2^"^^, so the overall distance between the reconstruction based 
on / and that based on I is diminishing (this is why /3 = 2^"^^ rather than /3 = 2^"^ was 
taken). Q.E.D. 

Consider now the Wyner-Ziv problem with perfect feedforword on the past source sym- 
bols at the decoder. Assume: 
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1. {Yi} is an arbitrarily distributed side- information signal available only at the decoder. 

2. {Xi}, the source signal, is given by Xi = Y^ + Ni, where {iVj} is i.i.d. Af{0,a'^), 
independent of {Y^}. 

Consider next the following scheme for this setting: 

• Encoder: operate exactly as encoder associated with (6). 

• Decoder: 

1. Add Ei=2 V/?^ - Ip-^'^+'^^Yk + /S-^Fi to received Y. 

2. Input the result into the decoder using {Ni} as the feedforward sequence (which 
is possible since at time i is revealed, and is of course known). 

3. Let the reconstruction be given by = Yi + Ni, where iVj is output of the 
decoder from the previous stage. 

Claim 2 As I ^ oo, the distortion of the scheme described converges to a'^2~'^^^~'^^\ 

Proof: Since 



k=2 

I 



k=2 



k=2 

assuming Y G [—A/2, A/2], the quantization resolution implies |y — y| < A/M = 2^(^~^)', 
so the input (index) given to the decoder in the second stage is within 1 from what it 
would have received had encoding been performed (with scheme in (6)) directly on the 
{Ni} sequence. Claim 1 implies then that the distortion between {Ni} and {Ni}, hence 
also between {X^} and {Xi}, is essentially a^2~^^^''^^\ It only remains to argue that our 
assumption Y G [—A/2, A/2] was justified. To this end, observe that: 



Var{y} < Var 1^ - ip-^'^+^^Yk + P'^yA 

U=2 J 



5](/32-i)/3-2M+/3- 

lk=2 



(7) 
(8) 



so as long as X^fe=2 -\//?^ — 'i-P'^'^'^^^Yk + P~^Yi has expectation and variance growing sub- 
exponentially with I, which is the case for all but the wildest processes, since A = 2'^, 
Pr{F € [-A/2, A/2]} is overwhelmingly smah. Q.E.D. 
Comments: 

1. The scheme is as simple as the channel coding one, with no binning required. 

2. This scheme achieves the conditional rate-distortion function for the case where the 
side information is available at both encoder and decoder, for an arbitrarily distributed 
side information process {Yi}. 

3. Observation 2, combined with the previous item, implies that for the regular Wyner- 
Ziv problem, in the case where the pairs {Xi,Yi) are i.i.d., with Y^ arbitrarily dis- 
tributed and Xi = Yi + TVj for Ni Gaussian and independent of Y^, there is no loss 
due to the absence of side information at the encoder. This fact can be deduced also 
directly from the single-letter expression. Indeed, the argument used in [13, Section 
3] to show that the Wyner-Ziv function coincides with the conditional rate distortion 
function when (Xj, Yi) are jointly Gaussian is readily seen to carry over to this more 
general case. 

4. The results of [12] can be shown to imply, for an arbitrarily distributed SI process {Yi}, 
and source given by Xj = Yi + Ni, for i.i.d. (but arbitrarily distributed) process {Ni}, 
that feedforward does not help for source coding with SI on both sides. Combined 
with the second item, this implies that the Wyner-Ziv performance in the presence of 
feedforward for an arbitrarily distributed SI process and source given by Xj = Yi + Ni, 
for Ni i.i.d. Gaussian, coincides with that for SI at both sides. Furthermore, we have 
just shown a simple scheme attaining optimum performance for this case which is no 
less simple than had the SI been available at the encoder as well. Thus, not only is 
there no loss for not knowing the SI at the encoder in terms of the fundamental limit, 
there is also no loss in the simplicity of the scheme attaining it. 

5. Non-causal dependence of decoding on the SI in the above scheme is only once, in the 
first step, for computing ^2^=2 VW^-^P'^'^'^^^Y^ + P'^Yi. The reconstruction in the 
remaining steps uses the SI causally. 
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3 Finite alphabets 

A. A Scheme for the Gel'fand— Pinsker Channel with Feedback 

Consider the finite-alphabet setting of the Gel'fand-Pinsker channel, as described in the 
introduction. Let S, U, X, Y have a capacity-achieving distribution, namely, a distribution 
achieving maxp^^i^) j[/(C/ ; Y) — I{U ; S)], where X = f{U, S). Consider the following scheme 
of coding with feedback for the Gel'fand-Pinsker channel^, building on the ideas of [1, 6]: 

• Transmitter: Maps the N message bits into the sequence U"''^, n\ = N/H{U\S), where 
C/"i is the output of the decoder corresponding to an optimal Slepian-Wolf encoder 
of ?7"i for side information S"'^ , when receiving the N message bits as input from the 
encoder and observing the side information 5"^ . 

• Sends through the channel, where Xi = f{Ui, Si), 1 < i < ni. 

• Channel: Corrupts according to x). 

• Receiver: Feeds channel output back to the transmitter. 

• Transmitter: Using F^i, compresses U'^'^ into niH{U\Y) new data bits. 

• Maps these bits into the sequence U!^^^i '^ , for n2 = niH{U\Y)/H{U\S), by letting 
U^^^i'^ be the output of the decoder corresponding to an optimal Slepian-Wolf en- 
coder of U for side information S (for n2-tuples), when receiving the niH{U\Y) new 
data bits as input from the encoder and observing the side information -5"^^"^ . 

• Sends X^^^^"^ through the channel, where X^ = f{Ui, Si), ui + 1 < i < ni + n2- 

• Channel: Corrupts X"^^^"^ according to p(j/|s, x). 

• Receiver: Feeds channel output Y^^_^i^ back to the transmitter. 

• Transmitter: Using Y^^^^^ , compresses U^^^i^ into n2H{U\Y) new data bits, 

and so on. After k iterations of this process, letting 1^ = J2i=ini, use a simplistic termi- 
nation code for conveying the n^-tuple Uj^ to the decoder, allowed to be based also 
on y}'' ,1 that will be available from the feedback. Thus, in effect, this termination code 

needs to communicate ~ nkH{U\Y) = N[H{U\Y)/H{U\S)]^ additional information bits. 
^Throughout, we ignore integer constraints, writing, e.g., N/H{U\S) rather than \N/H{U\S)~\. 
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Decoding: Let denote the decoder's estimated version of and let bk be 

the binary njki?([/|>S')-tuple obtained by taking the output of the Slepian-Wolf encoder (used 
at the A;-th stage of the encoding) when this n^-tuple is used as its input. Let C//*"^^_,_j^ be 
the conditional entropy decoding of an nfc_i-tuple of the source U given the corresponding 
nfc_i-tuple of Y as side information, for the Y sequence ^[.'^ ^+1 ^^"^ binary encoding 
bk- Now feed the njt_i-tuple ^/^'^J^+i Slepian-Wolf encoder used at the k — 1-th 

stage of the encoding, and let bk-i be the binary Uk-iH {U\S)-tuple obtained at its output. 
Continue this process for k iterations, until obtaining the binary iV-tuple bi, letting that 
be the decoded message bits. 

Analysis: The overall number of channel uses is 

N ^ 

+ ^ = jj^^ll\H{U\Y)/H{U\S)r'+L 

N l-[H{U\Y)/H{U\St 



H{U\S) 1 - [H{U\Y)/H{U\S)] 
N 

- H{U\S) - H{U\Y) ^ ' 

where L denotes the length of the termination code. In other words, assuming L <^ N, the 
number of information bits per channel use is essentially H(U\S) — H{U\Y) = I(U;Y) — 
I{U;S), the capacity. The probability of decoding error can readily be shown to diminish, 
taking k small enough so that the probability of an error in the Slepian-Wolf coding at any 
one of the k steps is negligible, yet large enough so that the length L of the termination 
code required to reliably transmit the last block (whose length decays exponentially with 
k) is negligible relative to N. 

B. Wyner— Ziv Coding with Feedforward 

Assume the Wyner-Ziv setting where source and SI are i.i.d. drawings of {X, Y). We further 
generate U according to Pu\x (so U — X — Y), and let X = f{U, Y), taking Pfj^x and / to 
be achievers of the Wyner-Ziv function. 

Shaping Subsystem: Given x"' which is Px-typical, a shaper Su^xi'i^"') is a 1-to-l 
mapping from {0, 1}"^(^I^) into T;7|x[ic"] (where denotes the set of u^-s that are 

jointly typical with x"). In other words, to every binary nH{U\X)-twple b there corresponds 
a (different) = Su^x(b,x'") such that {u",x'^) are jointly typical. Let S^^-^{-,x"') denote 
the inverse mapping of S^lx{■,x^). Existence of shapers follows from elementary facts 
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known from the method of types. Shaping systems can be implemented efficiently via 
arithmetic coding [6, 5]. 

Slepian-Wolf Coding: Given a typical u", let Cu{u^) denote the bit sequence of length 
nH{U\Y) resulting from an essentially optimal Slcpian-Wolf encoding of for the presence 
of side information Y"^ at the decoder. For h a binary sequence of length nH{U\Y) let 
C^^{b,y"') denote the reconstruction of the corresponding decoder when receiving b from 
the encoder and the side information sequence is y". 

Our Scheme: Fixing L, k, we take the length of the source sequence to be n = 
LEjZo [H{U\Y)/H{U\X)y. The following scheme builds on the ideas in [5]. 

Encoding: 

• Initialize T = 1, I = L, j = 1, and reverse the input so that X„_i, . . . ,Xi) 

• Take the block of source samples and generate a "noisy version" J/' by passing 
through the "channel" Pu\x- 

• while j < k do: 

• Do Slepian-Wolf encoding of U^+^ to obtain the binary lH{U\Y)-tuple b = Cu{U^^^). 
LetT = T + l + l,l = L[H{U\Y)/H{U\X)Y, U^+^ = Su\x{b, X^+^), and j =j + l 

• end while 

• return b = CuiU^""^) = Cu{U^_i) 
Decoding: 

• Initialize T = n, j = k — 1. 

• while J > do: 

• Let / = L[H{U\Y)/H{U\X)y and T = T-l. Construct by letting Xj = f{JJi, Yi) 
for each T < i < T + I, where U^+^ = C^^{b,Y^-^^). Obtaining via the 
feedforward, let b = Finally, let j=j-l 

• end while 

• return the reversed version of Xf 
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Performance: For k fixed and L large the Slepian-Wolf decoding is essentially error 
free, i.e., with high probability, at each of the k cycles of the while loop in the decoding 
C/i^"*"' = U^'^K Furthermore, the reconstruction X^^"*"' obtained at each of the k cycles 
is, with high probability, jointly typical with X^'^K Thus the overall distortion is, with 
high probability, approximately Ep{X, X). As for the rate note that, by construction, 
the number of bits emitted by the encoder is L[H{U\Y)/H{U\X)f~^ ■ H{U\Y), while 
the number of source samples encoded is n = L ^^H^ijJ^)/H(jj^j ^ Zi ■ Thus essentially, for 
K A; < L, the rate achieved is ii! !^ H{U\Y) - H{U\X) = I{U; X) - I{U; Y), the optimal 
Wyner-Ziv rate. 
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